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ABSTRACT 

The concept of Delphi technique was presented in a 
computer-assisted instruction (CAI) module designed for educational 
administration classes or inservice training of administrators. 
Instructional Dialogue Facility (IDF) Author Language on a 2000F 
Hewlett-Packard time-sharing system was used to write the sequence. 
Instructional objectives, simulated exercise, tutorial material, and 
a pretest and posttest were developed. The module was field tested by 
JT graaiiate students erirbllea in a 

course, 16 of whom had no prior content knowledge. Results showed 
that the module actually produced learning as determined by pretest 
and posttest. Learning also indicated a favorable attitude toward 

e Atii— It—was-s ugges t ed- that -t hris— module-couldr-serve-ather—uses-a^nd 

further development and testing of other such modules would be 
warranted. (Author/SC) 



4e4c***«**««««««*«««****************«****** ♦♦♦♦♦♦♦♦♦♦♦ 

* Documents acquired by ERIC include many informal unpublished * 

* materials not available from other sources. ERIC makes every effort * 

* to obtain the best copy available. Nevertheless, items of marginal * 

* reproducibility are often encountered and this affects the quality * 

* of the microfiche and hardcopy reproductions ERIC makes available * 

* via the ERIC Document Reproduction Service (EDRS) . EDRS is not * 

* responsible for the quality of the original document. Reproductions * 

* supplied by EDRS are the best that can be made from the original. * 



Development of a 
Computer-Assisted Instruction 
(CAI) Program on the Delphi Technique 
October 28-30, 1976 

by 

Carol McCurdy 



•PERMISSION TO REPRODUCE THIS COPY. 
R'GHTED MATERIAL HAS BEEN GRANTED BY 

■^Garol^McCur^ 



TO ERIC AND ORGANIZATIONS OPERATING 
UNDER AGREEMENTS WITH THE NATIONa' IN- 
STITUTE OF EDUCATION FURTHER REPRQ. 
DUCTtON OUTSIDE THE £R,C SYSTEM RE 
^';j[^/^«^'SSION 0^ THE COPYRIGHT 



US DEPARTMENT OF HEALTH. 
EDUCATION « WELFARE 
NATIONAL INSTITUTE OF 
EDUCATION 

THIS DOCUMENT HAS BEEN -REPRO- 
DUCED EXACTLY AS RECEIVED FROM 
-T HE-P E R^Ori-OR^ORjQANLZAnQNJJRiGlN^ 
ATING IT POINTS OF VIEWOR OPINIONS 
STATED DO NOT NEtESSARILY REPRE- 
SENT OF F iClAL NATIONAL INSTITUTEOF 
EDUCATION POSITION OR POLICY 



ABSTRACT 

The Delphi technique is a method for soliciting the informed 
opinions of selected experts and moving toward consensus through a 
series of questionnaires and feedback. An alternate instructional 
mode was needed for presenting Delphi technique concepts to people 
concerned with the administration of professional personnel. This 
was evidenced by the opinions of learners, observations of professors 
and the lack of computer-assisted instruction (CAI) materials on the 

topicT therefor^^ 

vide the means for people to learn about the Delphi technique, to 

"mak'e'appT^icytton 

the computer as it is used to assist in the educational process. 

The major problem of this study was to develop and test the 
effectiveness of a unique CAI module for use in the teaching of 
Delphi technique concepts. Two purposes were identified: first, 
to develop unique CAI material for use in educational administration 
classes or in-service training of educators; second, to provide an 
alternate instructional mode for presenting Delphi technique concepts 

Research was conducted principally at The University of Iowa 
Computer-Assisted Instruction Research Laboratory, Instructional 
Dialogue Facility (IDF). Author Language available on ^ 2000F Hewlett- 
Packard time-sharing system was used to write the Delphi sequence. 



Tutorial and simulation CAI modes were used for development of an 
Instructional module. Instructional objectives for the module were 
developed, a simulated hypothetical Delphi exercise was created and 
tested, a pretest and posttest were devised, and tutorial material 
was written. 

Graduate students enrolled in educational administration courses 
participated in large-group testing of the module under conditions 
for which it was designed. Of the 33 participants, 16 indicated 
having had no prior knowledge of Delphi technique concepts. Learning 
effectiveness as a result of the Delphi module was determined through 
use of a pretest and posttest. Statistical and summary data support 
this* statewnt' that' expos 

result in learning. In addition, learners indicated that it was an 
enaoyable-experteneey-rei^i^farcTng-the-nchtlan-tha^t— peop^^ 
like CAI. Comments, suggestions, and item statistics were considered 
most valuable in that they offered the greatest source of information 
for module analysis and revision. 

The Delphi module could be used as a tool at meetings and 
conferences and for short courses, workshops, and courses in other 
subject matter areas. Results of this study imply that further 
effort in developing and testing additional CAI modules is warranted. 
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OVERVIEW OF THE DELPHI TECHNIQUE 

The Delphi technique is a method for soliciting the informed opinions 
of selected experts and moving toward consensus through a series of question- 
naires and feedback. It is one of the most flexible projection techniques 
available today. 

Key characteristics of the Delphi technique are: 1) anonymity of 
survey panel members, the eliciting of individual and private responses 
from the participants to reduce the effects of socially dominant individ- 
uals; 2) statistical analysis; and 3) controlled feedback, which consists 
of sending iteration summaries to each participant. (3) 

Through~use""ofT^ 

eliminating face-to-face confrontation. Its goal is "to collect judgments 
and establish consensus about future probabilities in terms of such 
variables as time, quantity, and/or the des'raiility of some future 
state." (6) Contact with the experts is g n , lly made through a series 
of questionnaires and feedback designed to elicit progressively more 
carefully considered group opinions. 

COMPUTER-ASSISTED INSTRUCTION (CAI) 

Educational applications of the computer have multiplied rapidly during 
the past few years. Not only is the computer being utilized for administrative 
applications (such as computerized scheduling, mark and attendance reporting, 
payroll, inventory), but the computer is applied increasingly to instruction. 
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The interactive capacity of the computer tends to monopolize the 
attention and to encourage full involvement of learners of all ages and 
grade levels. Computer use can bring to the educational process such 
attributes as absolute patience and objectivity, immediate feedback, 
extended dissemination availability, and student-paced individualized 
instruction programs. (7) 

While some educators hesitate to experiment, research indicates that 
many others experience success and satisfaction with computer use. Indeed, 
effective computer utilization in the teaching-learning process includes 
examples representing practically all subject matter areas. (2,5,10) As 
implementation of computer applications continues to be supported where 
f eas i b 1 e ^. en 1 ar5.ed number s of . educa to r s b e c n and cornm i tt e d 

to computer-assisted instruction (CAI). 

~" PROBLEM/PURPOSES" OF'THE DELPHrSTUDY 

As evidenced by the opinions of learners, observations of professors, 
and the lack of computer-assisted instruction (CAI) materials on the topic, 
an alternate instructional mode was needed for presenting Delphi technique 
concepts to people concerned with the administration of professional 
personnel. Therefore, it was desirable to develop a CAI module to provide 
the means for people to learn about the Delphi technique, to make applica- 
tions to personnel, and at the same time to experience interaction with the 
computer as it is used to assist in the educational process. 

The purposes, then, of the Delphi study wer-*: first, to develop unique 
CAI material for use in educational administration classes or in-service 
training of administrators; and second, to provide an alternate instructional 
mode for presenting Delphi technique concepts. 
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GENERAL PROCEDURES 

Research was conducted principally at The University of Iowa Computer- 
Assisted Instruction Research Laboratory. Tutorial and simulation CAI modes 
were used for development of an instructional module on the Delphi technique. 

Tutorial instruction approximates teacher-student interaction; it is 
used to introduce a concept and to develop the student's skill in using it. 
Tutorial CAI material consists of a series of instructional frames. It is 
similar to printed programmed instruction except that the student works at 
a computer terminal and his/her responses are transmitted directly to the 
computer, which facilitates instantaneous and automatic feedback. After 
each response, the student can be branched to new instructional material 
wh-lct^ls.. especially, designed^^^^t^^ - 

Computer-based simulation is "a technique for studying the behavior 

of complex systems b y using a computer to manipulate the v ariables and 

observe the behavior of the model." (1) It is a hypothetical representa- 
tion of situations occurring in the real world. Typically, the computer 
presents the learner with information about a real situation; the learner 
is then required to make some kind of decision based on this information. 
The computer examines the decision and presents a new set of facts, and 
the learner is required to make more decisions. This cycle is followed 
throughout the computer-based simulation exercise. (10) Essentially, the 
student learns by being placed in a situation where s/he must make decisions 
to achieve certain goals. Insight is gained by (a) examining and manipu- 
lating variables in a setting that simulates situations in real life and 
(b) assessing the consequences of the decisions made. (7) 
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After investigating the use of several computer programming languages, 
it was decided that Instructional Dialogue Facility (IDF) Author Language, 
which was available on a 2000F Hewlett-Packard time-sharing system, would 
be used to write the Delphi sequence. IDF's simplicity and potential for 
classroom use were key reasons for its selection. 

Instructional objectives for the module were developed, the simulated 
Delphi exercise was created and tested, the pretest and posttest were 
devised, and the tutorial material was written. During the development 
period, several individuals made valuable comments which facilitated early 
module revisions. Upon completion of.the program, individual testing took 
place. Revisions were made according to the author's personal observations 
arKt- t!>e^ perf c>rnja^rvsfr 

Field testing was then conducted and revisions were again made, readying 

the program for . c1 assroom use . 

Graduate students enrolled in educational administration courses 
participated in large-group testing of the module under conditions for 
which it was designed. Of the 33 participants, 16 indicated having had 
no prior knowledge of Delphi technique concepts. Learning effectiveness 
as a result of the Delphi module was determined through use of a pretest 
and posttest. Statistical and summary data support the statement that 
exposure to the Delphi CAI course actually did result in learning. In 
addition, learners indicated that it was an enjoyable experience, rein- 
forcing the notion that people in general like CAI. Comments, suggestions, 
and item statistics were considered most valuable in that they offered the 
greatest source of information for module analysis and revision. 
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DEVELOPMENTAL PROCEDURE 

Figure 1 shows the general instructional sequence developed in the 
Delphi technique module. More specific structure.for development of the 
Delphi course was provided, with some modification, by the instructional 
model designed by Walter. (9) The 12 basic instructional design steps 
followed in the Delphi module development are represented in Figure 2: 

1) A topic (the Delphi technique) was selected based on requirements 
of the target population and on needed understanding of the Delphi method. 

2) The medium (CAI) and language (IDF) were selected for the purpose 
of communicating the instructional message to the learner. 

3) Instructional objectives were stated. 

in the instructional objectives. 

5) The pret est and postte st were assemble d from crit erio n items which _ 

were developed to measure whether or not an individual had acquired the 
specific behaviors described in each objective. 

6) A flowchart illustrating the general instructional sequence was 
drawn showing every possible path a student could follow through the 
lesson segment and indicating the conditions which determine each path. 

7) The learning activities designed to provide the student with the 
skills and knowledge necessary to attain each objective were developed and 
programmed using Instructional Dialogue Facility (IDF) Author Language. 

8) Several people from the target population tested the lesson segment 
(small -group test); revision was made where necessary. 

9) Step 8 was repeated. 
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10) The module was used with a group of 33 students from the target 
population under the conditions for which it was developed to determine 
if it teaches what it purports to teach (large-group test)* 

11) Results of the testing procedure were analyzed. 

12) Additional module revision was made. 

While the foregoing steps appear prescriptive, module revision was 
never-ending. Numerous revisions were made throughout the development 
process; more will ur^doubtedly be made in the future. 

IMPLICATIONS 

From the review of literature, it is apparent that use of the computer 
in the classroom is rapidly expanding. In practically all subject matter 
areas, effective computer utilization has been documented. Research indi- 
cates successful and satisfactory experience with computer applications at 
all levels of education. 

The Delphi technique is a topic that is difficult to cover in a classroom 
setting because of its combined complexity and generality. The module allows 
learners to proceed through entire development of the concept without inter- 
ruption. The simulated Delphi exercise was designed to Introduce the whole 
picture before the relating of specifics. By repeating the simulation follow- 
ing instruction, immediate application of Delphi technique concepts reinforced 
the learning of lesson objectives. 

As with any worthwhile teaching endeavor, a project of this nature 
requires a considerable amount of fortitude, a great deal of time, and a 
tremendous outlay of effort. Perhaps such a project could best be tackled 
by a team with each member contributing to the development of the final 
product. In order for development of this kind of material to be justified 

10 
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in terms of time and/or cost effectiveness, wide module use must occur in a 
number of different situations and places. 

The Delphi study strongly reinforced the notion that people in general 
like CAI. Exposure to the Delphi module was an enjoyable experience; 
learners felt that they had gained real knowledge and that going through 
the course was considered time well spent. Participants appreciated the 
presentation of a maximum amount of information in an acceptable amount of 
time. People were able to learn effectively without rote memorization, 
feeling belittled, or undue repetition. 

Several learners indicated that the Delphi module was helpful to them 
and that they would recommend it to others. Some individuals even suggested 

desired at meetings and conferences, for short courses, workshops, courses 
_in other subject matter areas and other universitie s, and for q uick reference 
when studying for comprehensive examinations. 

Comments referring to the lack of tediousness and frustration associated 
with the Delphi module indicate a contribution toward the alleviation of fear 
of the comDuter among those unfamiliar with it and with its capabilities. 
Display of the learner's name at strategic points was an especially pleasing 
aspect of the Delphi module. Participants expressed surprise and delight, at 
this personal touch from what they considered to be an impersonal computer. 

Some of the specific implications suggest that further effort in the 
developing and testing of other such modules is warranted. Several additional 
administrative tools are compatible with possible instructional presentation 
via CAI. Another interesting approach would be to test the module among 
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groups of people with non-educational areas of interest, as well as areas 
of interest in education but outside that of educational administ:"ation. 

In conclusion, this study represents a contribution toward positive 
CAI learning experiences and, therefore, helps to eliminate obstacles to 
effective future use of the computer for instruction. Experiences sucl as 
this provide involvement with CAI; the potential commitment to CAI follows. 
By developing unique CAI material and providing an alternate instructional 
method for presentation of Delphi technique concepts, the stated goals of 
this study were accomplished. 
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FIGURE 1. INSTRUCTIONAL SEQUENCE FLOWCHART 
(FOR THE DELPHI CAI MODULE) 
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FIGURE 2. BASIC INSTRUCTIONAL DESIGN MODEL 

. . (FOR THE DELPHI CAI MODULE) 
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